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ABSTRACT 



In 1982 and 1983 the Measurement and Evaluation 



Center of the University of Texas at Austin conducted a set of three 
studies analyzing student performance in a physics course required 
for engineering students (Physics 303K.) in relation to the students' 
scores on the College board Mathematics Level 1 and Physics tests and 
to an introductory physics course (Physics 306). The third of these 
studies, an investigation of whether the early course, Physics 306, 
helps overcome weak academic preparation and to determine the score 
intervals of the Achievement Test in mathematics to be used in 
placement was repeated in 1991 to update decision foundations. The 
1,?97 students who met the requirements for inclusion in this study 
were divided into those who had no previous college physics course 
before taking Physics 303K and those who has taken Physics 306. The 
introductory course appeared to be fulfilling its function of 
providing remediation and improving the chances of success for 
incoming engineering students. Certain score levels on the 
mathematics achievement test were recommended for placement 
decisions, with those scoring below 60U required or strongly advised 
to take the introductory course, and those scoring above 720 advised 
to enroll in the higher-level course. Four tables and one figure 
present study findings . (SLD) 
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VALIDITY STUDY OF THE COLLFXiE BOARD ACHIEVEMENT TEST 
IN MATHEMATICS FOR USE IN COURSE PLACEMENT 
IN ENGINEERING PHYSICS AT UT AUSTIN, 1991 

Nancy H. Bene, Nellie S. Cheng, 
Lynn M. Trent, and Barbara G. Dodd 

In 1982 and 1983, the Measurement and Evaluation Center (MEC) of The University of 
Texas at Austin conducted a set of three studies analyzing student performance in Physics 
303K in relation to the students' scores on two College Board Achievement Tests (Mathematics 
I,evel I and Physics) and to an introductory Physics course (Physics 306). Senior faculty- 
members of the Department of Physics had requested the assistance of the MEC because of the 
high rate of student failure - estimated to be approaching 50% - in Physics 303K, a required 
course for engineering students during their first semester at the University. The three studies 
and their results are reported in the Measurement and Evaluation Center Research Bulletin, RB- 
83-12, December 1983. 

The purpose of the third study was to investigate whether Physics 306 helps overcome 
weak academic preparation in high school mathematics and to determine the score intervals of 
the College Board Achievement Test in Mathematics Level I to be used in advising engineering 
students whether to register for Physics 306 prior to enrolling in Physics 303K. MEC staff 
members concluded that the introductory course, Physics 306, appeared to successfully reduce 
the effects of weak preparation in mathematics. Seventy-one percent of the students whose 
Mathematics Level I test scores were in the 200 to 590 range and who enrolled in Physics 306 
before taking Physics 303 K made satisfactory grades in Physics 303K, compared to 53% of 
the students who had similar test scores and who enrolled only in Physics 303 K. It was 
therefore inferred that Physics 303K appeared to have saved 18% of the low scoring students 
from failure. 

According to the findings of the third study, the MEC recommended that Mathematics 
Level I test scores be used for placement of engineering students into Physics 303K or into a 
sequence of Physics 306 followed by Physics 303 K. Students whose test scores ranged 
between 200 to 590 should be required (or strongly advised) to enroll in Physics 306 before 
taking Physics 303 K; students with scores from 600 to 680 should make their own enrollment 
decisions regarding Physics 306 and/or Physics 303K; students whose Mathematics Level I 
test scores were 690 and above should be advised to enroll in Physics 303K. 
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The MEC staff members presented these results and recommendations to the 
representatives of the Department of Physics, who in turn informed the Dean of the College of 
Engineering. The faculty members of the College of Engineering were urged to advise all 
incoming engineering freshmen who scored below 600 on the Mathematics Level I test to 
register for Physics 306 before taking Physics 303K. Furthermore, after consultation with the 
College of Engineering, the Department of Physics decided to offer Physics 306 during the 
second as well as the first summer term in order to allow students less well prepared in 
mathematics another opportunity to improve their skills and to increase their chances of 
succeeding in the engineering curriculum. 

From the Fall Semester 1984 to the present time, the official Course Schedules of the 
University of Texas at Austin have contained the following statement preceding the Department 
of Physics course listings: 

PHY 306 is a preparatory course for the PHY 303K/303L course 
sequence. Engineering students who have not taken high school 
physics, who have a weak mathematical background, or who score less 
than 600 on the College Board Achievement Test in Mathematics Level I 
are strongly advised to enroll in this course. 

In the spring of 1991, the Department of Physics faculty felt that Physics 306 provides 
appropriate remediation for the incoming freshman students who are inadequately prepared to 
take Physics 303K. Of the students who were enrolled in Physics 303K during one of the six 
semesters from the spring of 1989 to the fall semester 1990, only 20% failed. Even though the 
recommendation was strongly stated, less than 50% of the students whose scores were below 
600 on the Mathematics Level I test chose to enroll in Physics 306. The faculty members 
considered changing the recommendation so that Physics 306 would become a mandatory 
requirement for the students with low scores on the Mathematics Level I test. Given the fact 
that the current advising statement is based on 1982 data analyses, the Department of Physics 
requested that the MEC redo the third study conducted in 1983, using more recent data. 

Method 

Subjects 

Students who took Physics 303K at UT Austin in one of the six semesters from Spring 
1989 through Fall 1990 were designated as subjects if they (a) had enrolled in Physics 303K 
for the first time; (b) had prior credit for or concurrent enrollment in a .laboratory course 
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(Physics 103M); (c) had prior credit for or concurrent enrollment in Mathematics 408C, 308L, 
or 308K; (d) had taken the Mathematics Level I test; and (e) had received a letter grade (F - A) 
in Physics 303K. 

Physics 303K is a first semester lecture course in college physics for engineering 
students, although non-engineering students are also permitted to enroll in the course. It is a 
survey of physics, primarily covering the laws of motion, heat, and wave phenomena. Its 
prerequisites include high school study of physics; concurrent enrollment (or prior credit) in its 
laboratory complement, Physics 103M; and credit for (or concurrent registration in) calculus 
(Mathematics 408C, 308L or 308K). 

The 1,297 students who met the requirements for inclusion in the study were divided into 
two groups: (1) those who had taken no previous college physics courses prior to enrolling in 
Physics 303K; and (2) those who had taken the preparatory course, Physics 306, before 
enrolling in Physics 303K. The first group of students who enrolled immediately in Physics 
303K consisted of 1,033 students, while 264 students were members of the group that first 
took Physics 306 a d then enrolled in Physics 303K. 

Procedure . Mathematics Level I test scores for the first year engineering students were 
available from the University data files because that test is required for registration and 
placement in lower-division mathematics courses at UT Austin. 

The statistical investigations conducted in 1991 replicated the aptitude- treatment analyses 
performed on the 1982 data. Two separate regression analyses were conducted separately for 
each group of students. The Mathematics Level I test scores were used to predict grades in 
Physics 303K. Each regression analysis yielded a correlation (validity) coefficient (r) and a 
regression (prediction) equation. The validity coefficient indicated the correlation or degree of 
linear relationship between the Mathematics Level I test scores and the grades in 303K. The 
regression equation enables one to predict the grade in Physics 303K that could be expected of 
students with a given Mathematics Level I test score. 

Once the expected Physics 303K grades were calculated from the Mathematics I^vel I test 
scores by use of the two regression equations (one for each group of students), the Johnson - 
Neyman technique was employed to determine the Mathematics Level I test scores that were 
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associated with significant differences between the expected grades for the two groups. That 
is, the Johnson-Neyman technique revealed the score intervals on the Mathematics Level I test 
that could be used to decide if the engineering students should take Physics 306 before 
enrolling in Physics 303K. 



Table 1 presents descriptive statistics for the Mathematics Level I test scores and Physics 
303K grades for each of the two groups. The descriptive statistics include the means and 
standard deviations of the test scores and grades. 

The mean of the Mathematics Level I test scores for the Physics 303K group (640.86) 
was higher than the mean of the test scores for the Physics 306-303K group (536.78). The 
difference between these means is greater than the difference between the means of the two 
groups* scores reflected in 1982 data (Physics 306-303K, mean = 544.93; Physics 303K, 
mean = 622.44). This difference is most likely due to the fact that in 1982 the Mathematics 
Level I test scores were used only indirectly to advise students about enrolling in Physics 306 
or Physics 303K. 



Results and Discussion 



Table 1 



Descriptive Statistics for Students in the Course Sequence Groups; 
Scores on the Mathematics Level I Achievement Test 
and Course Grades in Physics 303K 



Mathematics 
Level I Test Scores 



Physics 303K 
Grades 



Course 
Sequence 



Mean 



Standard 
Deviation 



Mean 



Standard 
Deviation 



Physics 306-303K 264 536.78 62.71 2.08 1.14 

Physics 303K 1033 640.86 65.45 2.46 1.21 
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The mean Physics 303K grades presented in Table 1 show that the Physics 303K group 
had a higher mean grade (2.46) than did the Physics 306-303K group (2.08). This pattern is a 
reversal of the direction of the difference shown by the 1982 data. However, the mean grades 
for both 1982 groups (Physics 306-303K, mean = 2.16; Physics 306, mean = 2.07) were 
similar to the level of the present Physics 306-303K group. The higher mean grade of the 
present Physics 303K group may reflect the higher ability levels as reflected in Mathematics 
Level I test scores that were higher for the Physics 303K group in the present data than in the 
1982 Physics 303K group. 

Table 2 presents the validity coefficients and the regression equations that were obtained 
from the regression analyses. The validity coefficient obtained for the Physics 303K was 42; 
it indicates that students who received high scores on the Mathematics Level I test tended to 
perform well In Physics 303K, while students who received low test scores tended to perform 
poorly Ln Physics 303K. The validity coefficient of .42 may be considered high enough to 
justify the use of Mathematics Level I test scores to predict the success of students in Physics 
303K. 

The validity coefficient of .30 that was obtained for the two -course sequence group is 
somewhat low; it indicates a moderately weak relationship between the Mathematics Level I test 
scores prior to taking Physics 306 and the grades students received when they later took 
Physics 303K. This relationship shows that the Mathematics Level I test score is not as good a 
predictor of Physics 303K grades if the students take Physics 306 before Physics 303K. The 
validity coefficient of .30 indicates that students who have low scores on the Mathematics 
Level I test may be helped by taking Physics 306 prior to taking Physics 303K. 

The two regression equations presented in Table 2 were used to calculate the expected 
grades for students in each of the two course sequence groups. The regression lines shown in 
Figure 1 graphically depict the expected grades obtained for each of the groups. As can be 
seen from Figure 1, the regression lines intersect at a Mathematics Level I test score of 
approximately 720. For students with Mathematics Level I test scores below 720 (the point of 
Intersection), the regression lines indicate that students in the Physics 306-303K group are 
expected to receive higher Physics 303K grades than students in the Physics 303K group. 
Conversely, for students with Mathematics Level I test scores above 720, students in the 
Physics 303K groups are expected to receive higher grades in Physics 303K than students in 
the Physics 306-303K group. 
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Table 2 

Results of Regression Analysis for Each of the Course Sequence Groups; 
Regression Equations and Coefficients of Correlation 
Between Mathematics Level I Test Scores and 
Course Grades in Physics 303K 



Course 
Sequence 


Regression Equation 




Coefficient 
of 

Correlation 


Physics 


Expected Grade 






306-303K 


in PHY 303K = (Math I test score x .0054) - 


.83383 


.30 


Physics 


Expected Grade 






303K 


in PHY 303K = (Math I test score x .0078) - 


2.56093 


.42 



In order to determine the Mathematics Level I test scores that correspond to statistically 
significant differences between the expected Physics 303K grades for the two groups, the 
lohnson-Neyman technique was employed. This analytical technique revealed that the exact 
point o; intersection of the two regression lines was 718.35 and that, for Mathematics Level I 
test scores lower than 620.88, there was a statistically significant difference between the 
expected Physics 303K grades for the two groups; that is, students with Mathematics Level I 
test scores lower than 620.88 can, with reasonable confidence, be expected to perform better in 
Physics 303K if they take the two-course sequence than if they enroll directly in Physics 
303K. 

In addition, the analysis revealed that for students with Mathematics Level I test scores 
higher then 718.35, the differences in the expected grades for the two groups were not 
statistically significant. Thus, the expected grades for students in the two course sequence 
groups were significantly different only for Mathematics Level I test scores that were lower 
than 620.88 - for practical purposes, 620 or lower. 

Based on the results of the Johnson-Neyman analysis, the Mathematics Level I test scores 
were divided into three intervals that could be used to place engineering students into the two- 
course sequence, Physics 306-303K, or to advise them to directly enroll in Physics 303K. 
The first interval included Mathematics Level I test scores of 200-620, because the differences 
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in the expected Physics 303K grades for the two groups were significant for Mathematics 
Level I test scores lower than 620,88. Students with Mathematics Level I test scores that are in 
this interval should be advised to take Physics 306 prior to taking Physics 303K because 
students with mathematics skills at that level who have taken Physics 306 are expected to 
receive higher grades in Physics 303K than students who have not taken Physics 306. 

The second Mathematics Level I test scores interval includes test scores of 630-710 
because the intersection point of the two regression lines (718.35) lies between the two scores 
of 710 and 720. While the students with Mathematics Level I test scores within this interval 
were expected to receive higher grades in Physics 303K if they had taken the two-course 
sequence rather than enrolling directly in Physics 303K, such students probably should be 
permitted to choose whether or not to take Physics 306 prior to Physics 303K because the 
Johnson-Neyman technique failed to find statistically significant differences between the 
expected Physics 303K grades for the two groups. For advising purposes, the proximity of a 
student's Mathematics Level I test score to the lower and upper limits of their score interval 
should be taken into consideration when advising a student whether or not to take Physics 306 
prior to Physics 303K - that is, if a student's Mathematics Level I test score was closer to 630 
than 710, he/she should perhaps be advised to take Physics 306 prior to Physics 303K, while a 
student with a Mathematics Level I test score closer to 710 should perhaps be advised to take 
Physics 303K as his/her first physics course. 

The third Mathematics Level I test score interval includes test scores that range from 720 
to 800 because, unlike the other two intervals, students in the Physics 303K group could be 
expected to receive equal or higher grades in Physics 303K than the students in the Physics 
306-303K groups, even though the Johnson-Neyman technique revealed that the differences in 
the expected grades for this score interval were not statistically significant. Given the direction 
of the expected grade differences and the fact that the differences were nonsignificant, students 
whose Mathematics Level I test scores are in this interval should be advised to take Physics 
303K as their first physics course. 

While the Johnson-Neyman technique yielded information concerning the significant 
differences between the expected grades fur the two groups and the two Mathematics Level I 
test scores that divided the test scores into three intervals, the technique did not indicate how 
the students in each score interval for each course sequence actually performed in Physics 
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303K. Therefore, the students in each course sequence group were assigned to the score 
intervals recommended by the present analysis in order to inspect the relationship between the 
recommended score intervals and the students' actual performances (letter grades) in Physics 
303K. The grades of the students in each score interval for each course sequence then were 
collapsed into two categories, Unsatisfactory (grades of £ and D) and Satisfactory (grades of 
£, B, and A) performance in Physics 303K. Table 3 presents the number and percentages of 
students in the Unsatisfactory and Satisfactory grade classification for each score interval 
within each course sequence. 

The numbers expressed as percentages in the set of columns labeled "Physics 303K 
Grades" are the percentages of students in a given score interval for a. particular course 
sequence who received unsatisfactory grades and satisfactory grades in Physics 303K, 
respectively. For example, 26% (N = 61) of the 238 students in the two-course sequence who 
earned Mathematics Level I test scores lower than or equal to 620 received unsatisfactory 
grades (E, Q) in Physics 303K, while 74% (N = 177) of the students in the two-course 
sequence who earned Mathematics Level I test scores lower than or equal to 620 received 
satisfactory grades (£-A) in Physics 303K. 

Each number expressed as a percentage in the column labeled "Total" is the percentage of 
students in the two course sequences whose test scores placed them in one of the three 
Mathematics Level I test score intervals. For instance, 90% (N = 2'38) of 264 students in the 
Physics 306-303K group had received Mathematics Level I test scores lower than or equal to 
620. 

Tables 3 makes it possible to compare the percentage of students who were in each score 
interval for each of the course sequence groups; these data enable one to assess the relationship 
between the score intervals and the students' actual performances in Physics 303K. For the 
200-620 score interval, a comparison of the percentages of students who earned satisfactory 
grades in Physics 303K for each of the course sequence groups reveals that 74% of the 
students in the two-course sequence group received satisfactory grades in Physics 303K, while 
69% of the corresponding students who took only Physics 303K received satisfactory grades. 
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Table 3 



Relationship Between Mathematics Level I Test Score Intervals 
and Actual Physics 303K Grades for Each of the Two Course 
Sequence Groups, Physics 306-303K and Physics 303K 



Course 


Score 


Physics 303K Grades 




sequence 


Interval 


HE 


C-A 


lotal 




720-800 


0 

0% 


2 

100% 


2 

1% 


Physics 306-303K 


630-710 


2 

8% 


22 
92% 


24 
9% 




200-620 


61 

26% 


177 
74% 


238 
yv /o 




720-800 


4 

3% 


136 
97% 


140 
14% 


Physics 303K 


630-710 


74 
15% 


422 
85% 


496 
48% 




200-620" 


124 
31% 


273 
69% 


397 
38% 



For the 630710 score interval on the Mathematics Level I test, a higher percentage of the 
students in the Physics 306-303K group (92%) received satisfactory grades than the 
corresponding students in the Physics 303K group (85%). The difference between these two 
percentages reflects the higher performance of students in the two-course sequence than that of 
students who did not take the introductory course. 

For the highest Mathematics Level I test score interval (720-800), nearly optimum 
percentages of the students in the 303K group (97%) and the two-course sequence group 
(100%) received satisfactory grades in Physics 303K. Any comparison of these percentages 
should be made with caution because only two students in the 720-800 interval elected to take 
the introductory course, Physics 306. 

In general, the findings of the percentage analyses of the actual grades reveals the same 
conclusions that were drawn from the results of the Johnson-Neyman analysis - that is, 
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engineering students with Mathematics Level I test scores lower than or equal to 620 should be 
required (or strongly advised) to take Physics 306 prior to enrollment in the required Physics 
303K course; engineering students with Mathematics Level I test scores in the 630- 710 score 
interval should be allowed to select whichever course sequence they believe would better serve 
their needs; engineering students with Mathematics Level I test scores higher thin 710 should 
be advised to enroll in Physics 303K as their first physics course. 



The introductory course (Physics 306) appears to be fulfilling its function of providing 
remediation and improving the chances of success for incoming engineering students who are 
inadequately prepared to take Physics 303K. One indication of its success is given by the 
correlation coefficients in Table 2. The relationship between Mathematics Level I test scores 
and Physics 303K grades is lower for the two-course sequence group (r = .30) than for the 
Physics 303K group (.42). The difference between these correlations indicates that the 
introductory course (Physics 306) appears to reduce the effects of weak preparation in 
mathematics. A second indication of its effect on Physics 303K performance is to be found in 
Table 3, which shows that higher percentages of students in each of the three Mathematics 
Level I test score intervals earned satisfactory grades when they took Physics 306 before 
enrolling in Physics 303K. 

On the basis of the results from the present study, MEC staff members recommend that 
scores on the Mathematics Level I test be used for placement of engineering students into 
Physics 306 or Physics 303K and that the following set of scores be used for placement 
decisions. 



Recommendations 



T est Scores 
200-620 



Placement 

Student to be required (or strongly advised) to enroll 
in Physics 306 before enrolling in Physics 303K. 



630-710 



Student to be advised to make his/her own 
enrollment decision. 



720-800 



Student to be advised to enroll in Physics 303K. 



However, because a score of 610 on the Mathematics Level I test is the minimum score 
required to earn credit by examination in Mathematics 305G with a grade of B, MEC staff 
members modified the first two recommended score intervals to 200-600 and 610-710. 
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Percentages nearly identical to those shown in Table 3 were obtained in an MEC re-analysis of 
the data, which used score ranges of 200-600, 610-710, and 720-800. (See Table 4.) Based 
on this data, MEC staff members recommend the following set of score intervals be used for 
placement of engineering students into Physics 306 or Physics 303K. 

Test Scores Placement 

200-600 Student to be required (or strongly advised) to enrol! 

in Physics 306 before enrolling in Physics 303K. 

610-710 Student to be advised to make his/her own 

enrollment decision. 

720-800 Student to be advised to enroll in Physics 303K. 
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Relationship Between Mathematics Level I Test Score Intervals 
and Actual Physics 303K Grades for Each of the Two Course 
Sequences: Re-Analysis Using Modified Score Ranges 



Course 


Score 


Physics 303K Grades 




Sequence 


Interval 


D,E 


Q-A 


Total 




720-800 


0 

0% 


2 

100% 


2 

1% 


Physics 306-303K 


610-710 


2 

6% 


32 
94% 


34 
13% 




200-600 


61 

27% 


167 

73% 


228 
86% 




720-800 


4 

3% 


136 
97% 


140 
14% 


Physics 303K 


610-710 


101 
17% 


502 
83% 


603 
58% 




200-600 


97 

33% 


193 
67% 


290 
28% 




